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I. 


INTRODUCTION 


A. BACKGROUND 

The Department of Defense (DOD) accounts for a substantial percentage of the 
federal budget each year (Albano, 2013). Therefore, DOD acquisition professionals must 
focus on achieving best value for the government through pursuit of cost savings, while 
also adhering to stated regulations. In the past. Government Accountability Office (GAO) 
reports, professional journals, and acquisition magazines have identified a wide disparity 
within source selection processes; while some are best in class, others consistently fail to 
achieve the desired outcomes. For example, strengths of current source selection processes 
include the use of: engineering procedures in strategy definition. Integrated Project Teams 
(IPTs), Performance-Based Acquisition (PBA), and good communication techniques. On 
the other hand, risk assessments, source selection evaluation factors, schedule slips, and 
cost overruns are all areas of significant weakness (Maser & Thompson, 2013). In 1992, 
the GAO highlighted DOD contract management as high-risk; over two decades later, it is 
still a point of major concern (Albano, 2013). Moreover, the DOD inspector general 
identified 142 different deficiencies relating to contract management (Albano, 2013). 

A more recent GAO (2010) report highlighted training, education, and experience 
level of the acquisition workforce as areas of concern. Each of these topics represents an 
opportunity for improvement and savings, particularly initiatives focused on education and 
training. However, this report only highlights a fraction of the potential cost savings within 
federal acquisition. As such, GAO, DOD, and Air Force Installation Contracting Agency 
(AFICA) investigators have all engaged in a thorough analysis of previous procurements in 
an attempt to identify areas for future savings. AFICA is an Air Force organization 
designed to analyze purchasing data and ultimately develop strategies for strategic sourcing 
(R. Westermeyer, personal communication. May 15, 2015). Strategic sourcing is an 
institutional level of acquisition where products and services are purchased at the enterprise 
level to drive down costs through economies of scale, demand management, and supplier 
integration (R. Westermeyer, personal communication. May 15, 2015). 
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The steps of the contract management process, which can be optimized to produce 
cost savings, are: procurement planning, solicitation planning, solicitation, source selection, 
contract administration, and contract closeout (Garrett, 2007); however, our research 
focuses primarily on the activities prior to contract award. Furthermore, the source 
selection process can be broken down into two contracting methods: sealed bidding and 
negotiations. Typically, sealed bidding is reserved for simple, well-defined requirements. 
The alternative, contracting by negotiation, is utilized for those acquisitions in which 
requirements are not simple or well-defined. 

Contracting by negotiation is discussed in FAR 15.101 and is further broken down 
into two source selection methods. The first, tradeoff, allows the government the flexibility 
to weigh non-cost factors as part of an overall evaluation assessment. The second, lowest 
price technically acceptable (LPTA), requires the government to first rank all offers in 
reverse price order, then review them in that order until a technically acceptable offer is 
identified (FAR 15.101-2). Each acquisition is unique, and the appropriate method can only 
be selected after reviewing the complexity of the program and all pertinent risks (Shultz, 
2015). 

B. PROBLEM STATEMENT 

Currently, there is little empirical evidence identifying a relationship between 
source selection method and contract outcomes. This lack of reliable data on the subject has 
forced contracting professionals to rely heavily on experience and anecdotal evidence, 
which can lead to confusion. Consider, for example, the terminology involved in the 
discussion. Despite the specificity of the definitions provided in the Federal Acquisition 
Regulation (FAR), many top analysts—some of whom have published works on the 
topic—still confuse key terms. For example, tradeoff is often confused with best value, the 
overarching term describing the entire continuum of options available to contracting 
officers charged with ensuring greatest benefit to the government. This error is most 
prevalent in the private sector; however, even the Honorable Jacques Gansler—former 
Undersecretary of Defense for Acquisition, Technology, and Logistics (USDAT&L)— 
muddles the terms in a paper published for a U.S. Senate subcommittee (Levin, 2014). 
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Identification of this problem is not meant to be accusatory; instead, our aim is to recognize 
the need for empirical data that eliminates confusion on the subject of source selection 
methods and their related outcomes. 

C. PURPOSE STATEMENT 

The purpose of our research is to empirically identify key relationships between the 
source selection methods and measurable contract outcomes. The motivation for our 
research stems from the lack of empirical evidence regarding this relationship, as well as 
from a desire to improve contract outcomes. Unnecessarily long lead times, protests, vague 
specifications, and cost and schedule overruns all plague defense acquisition. Ideally, an 
acquisition should strive to keep procurement acquisition lead time (PALT) as low as 
practical and Contractor Performance Assessment Reporting System (CPARS) ratings as 
high as possible. Positive contract outcomes are rooted in the procurement planning phase; 
it is in this phase that the procurement method, source selection method, and contract type 
are decided. Our research discusses the relationship of outcome variables to the chosen 
source selection method, with the ultimate goal of better understanding the implications of 
this decision facing acquisition teams. 

D. RESEARCH QUESTIONS 

In this research, we answer the following questions: 

1. What is the relationship between the source selection method (LPTA versus 
tradeoff) and CPARS ratings? 

2. What is the relationship between the source selection method (LPTA versus 
tradeoff) and PALT? 

3. What percentage of procurements using the tradeoff source selection 
method is awarded based on non-price factors? 

E. RESEARCH BENEFITS AND LIMITATIONS 

The principle benefit of our research is that it provides data relating to source 
selection methods and their subsequent contract outcomes. We accessed contract files 
containing all data necessary for complete statistical analysis. Historically, there has been a 
lack of empirical data on this subject; our research and analysis addresses this gap. 
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The main limitation of our research is the sample size of our data. Time and travel 
constraints limited our data collection to include the contract files from only two Air Force 
bases, as we discuss further in Chapter III. To broaden the scope of the research and reach 
more definitive conclusions, we would ideally have been able to access more contract files 
from additional installations, including those from defense branches other than the Air 
Force. 

F. ORGANIZATION OF REPORT 

This first chapter presented the background, problem statement, purpose statement, 
research questions, and benefits and limitations of our research. In Chapter II, we provide a 
review of relevant literature regarding source selection methods, including guiding 
principles for the federal acquisition system, significant defense acquisition reform 
initiatives, an overview of the contract management process, pre-award acquisition 
decisions, and a description of each source selection method (LPTA and tradeoff) and their 
corresponding implications. In Chapter III, we explain the methodology used to conduct 
our research, which includes a discussion of Air Force contracting structure and systems 
together with a description of our data collection process and its unique elements. In 
Chapter IV, we describe our data, present the results, discuss our analysis, and make 
recommendations based on the findings. In Chapter V, we summarize our findings, present 
our conclusion, answer the research questions, and suggest areas for further research. 

G. SUMMARY 

This chapter served as an introduction to our research, providing a brief background 
of source selections and their role in the acquisition environment. It also stated the problem 
to be addressed, the purpose of our research, and the research questions to be answered. 
Lastly, we stated the benefits and limitations of the research. In the next chapter, we 
examine academic journals, GAO reports, and other source selection texts in order to 
review the literature and increase understanding of federal contract management. 
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II. LITERATURE REVIEW 


A. INTRODUCTION 

The purpose of this chapter is to present a review of the literature concerning the 
federal contract management process, specifically regarding the role of source selection 
methods available to contracting officers. To ensure a comprehensive understanding of 
all relevant sources, the review incorporates information from federal regulations, the 
Government Accountability Office (GAO), and prominent contracting journals. 

First, we cover the guiding principles for the federal acquisition system, followed 
by significant defense acquisition reform initiatives. Next, we discuss the government 
contract management process, before moving to pre-award acquisition decisions. Finally, 
we will explain the two source selection methods, lowest price technically acceptable 
(LPTA) and tradeoff, as well as implications of each. 

B. GUIDING PRINCIPLES FOR THE FEDERAL ACQUISITION SYSTEM 

Federal contracting is very different from commercial purchasing. A wide array of 
statutory and agency-instituted regulations restricts the actions of federal contracting 
officers, chief of which is the Federal Acquisition Regulation (FAR). This body of 
regulation provides instruction on the entire contract management process, from the 
initiation of procurement planning to the final steps of contract closeout. The FAR is a 
living document, which regulators still revise to this day, as new policies are introduced. 
However, regulators realize that the diverse scenarios contracting officers face far exceed 
the capacity of a single regulation. Therefore, the FAR instructs acquisition personnel to 
consider actions approved if not explicitly prohibited: 

In exercising initiative. Government members of the Acquisition Team 
may assume if a specific strategy, practice, policy or procedure is in the 
best interests of the Government and is not addressed in the FAR, nor 
prohibited by law (statute or case law). Executive order or other 
regulation, that the strategy, practice, policy or procedure is a permissible 
exercise of authority. (FAR 1.102(d)) 
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The vast size of the federal government necessitates additional regulations to help 
establish policy. Subordinate to the FAR are various agency supplements, with the DOD 
maintaining the Defense Federal Acquisition Regulation Supplement (DFARS), which is 
one of the more robust versions. Each of the service branches has added another level of 
policy below the DFARS, and most contracting activities have issued several layers of 
policy memoranda below each service-specific supplement. However, as our research 
focuses on defense procurement, our analysis does not extend to the level of detail 
beyond the DFARS, therefore we have not included service supplements. 

In addition to describing which actions are not permitted, the FAR explains the 
four basic principles of government procurement. We have listed each below. 

1. Satisfy the Customer in Terms of Cost, Quality, and Timeliness of the 
Delivered Product or Service 

Ultimately, the role of federal procurement is to augment organic government 
personnel and equipment in order to carry out the mission or function of the end user. In 
the DOD, the end user is typically the warfighter, and all acquisition actions should be 
structured to provide the necessary level of support for our nation’s warriors, whether in 
garrison or abroad. Additionally, the FAR expands this concept by directing contracting 
officers to (1) use commercial items whenever possible, (2) conduct business with 
vendors that can demonstrate competence, and (3) seek out competition. 

2. Minimize Administrative Operating Costs 

Awarding a contract to a vendor is only one step in the contract management 
process. Depending on the size of the purchase, several weeks or even months of market 
research and source selection procedures may have taken place beforehand. Likewise, 
modifications, terminations, and contractor surveillance tasks can extend years beyond 
the effective date of a contract. Government officials should take every measure to 
streamline the processes before and after award in order to remain good stewards of 
taxpayer dollars. 
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3. Conduct Business with Integrity, Fairness, and Openness 

Unlike profit-seeking firms, which are responsible only to shareholders or owners, 
federal institutions have additional mandates because of their reliance on taxpayer 
dollars. One such mandate is the requirement for transparency, which ranges from 
publicizing new solicitations online to fielding Freedom of Information Act requests on 
existing contracts. 

4. Fulfill Public Policy Objectives 

The final objective of government procurement is a set of policy agenda items 
established by Congress, which are not meant to directly affect the warfighter or the 
contracts themselves. The most prominent example is the small business set-aside 
program, which ensures that small, disadvantaged firms receive a portion of government 
contracts. In many cases, accommodating these goals often comes at the expense of price 
or schedule, although proponents argue that small companies are just as capable as are 
established defense firms. This objective of meeting Congressional agenda items through 
procurement was not an original goal of the federal government, but policymakers 
introduced and subsequently modified the objective later over the course of a series of 
acquisition reforms. The following is a brief summary of those reforms. 

C. SIGNIFICANT DEFENSE ACQUISITION REFORM INITIATIVES 

As the principles described in the previous section indicate, simultaneously 
achieving all federal procurement goals is typically not feasible. Instead, contracting 
officers frequently weigh different priorities against each other, resulting in a balance of 
results that vary among acquisitions. Originally, the government was chiefly concerned 
with receiving an efficient price in a generally equitable way (Manuel, 2011); over time, 
however, the objectives of public policy have changed. The following is a list of relevant 
legislation, along with an indication of which goal(s) they support. 

I. Armed Services Procurement Act of 1947 

The Armed Services Procurement Act (ASPA) of 1947 represents the DOD’s first 
real departure from competitive acquisition during peacetime operations. Starting in the 
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18th century, federal regulations called for open solicitation of federal requirements, 
relying on market forces to obtain the best price and safeguard taxpayer dollars. 
However, after experiencing the increased demands of both world wars, officials realized 
that some items contained rigid delivery or performance requirements that made 
competition problematic or outright infeasible (Manuel, 2011). The ASPA applied only 
to defense agencies and still called for competition, but it included 17 exemption 
categories, allowing contracting officers much more latitude in procuring critical items. 

2. Small Business Act of 1953 

While the Small Business Act (SBA) was passed in 1953, the Hoover 
administration had laid the foundation more than two decades earlier (Bail, 2010). After 
the stock market crash of 1929, President Hoover formed the Reconstruction Finance 
Corporation to loan government funds to businesses, with the hope of jump-starting the 
economy. Over the next 24 years, the organization continued to evolve, giving rise to the 
Small War Plant Corporation and Small Defense Plant Corporation during World War II 
and the Korean War, respectively. The focus of the effort shifted from stimulating the 
economy to retaining wartime production capacity to ultimately guaranteeing that small 
businesses remained competitive in future government contracts. 

Interestingly, the SBA has changed relatively little in the 62 years since its 
passage, considering the number of changes made to competition and commercial 
acquisition policy. For example, the original act provided for small business size 
standards that were specific to each agency, formed the Small Business Administration to 
help small business concerns, and focused on awarding contracts to small businesses, 
“even if awards were made at a higher rather than lower price” (Bail, 2010, p. 80). One 
significant revision came in 1978 when the federal government opted to reserve awards 
below $25,000 to small businesses—the precursor to today’s small business set-asides. 

3. Truth in Negotiations Act of 1962 

After passing the ASPA, Congress closely monitored the effectiveness of 

negotiated contracts. During the 1950s, several GAO reports confirmed the military had 

overpaid by approximately $61 million in a five-year span (Williams, 1970). While the 
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DOD initially proposed an internal regulation scheme, Congress overruled the DOD’s 
plan and passed the Truth in Negotiations Act (TINA), requiring contractors to submit 
and certify cost or pricing information when proposals exceed a specific threshold. The 
dollar limit has changed several times over the last several decades, as well as specific 
provisions of the law itself (Calhoon & Sybert, 2012). However, the intent of the law 
remains the same as when originally passed—ensuring government negotiators have the 
required information to effectively negotiate with contractors. 

4. Competition in Contracting Act of 1984 

While the TINA helped improve negotiated contracts, legislators still felt that a 
return to open competition was in the best interest of the government. However, instead 
of returning to total competition, drafters agreed on full and open as opposed to maximum 
competition (Manuel, 2011). The former term describes competition as contracting 
officers understand it today: reasonable attempts to include vendors capable of 
performing as required in the contract. Maximum competition, an all-out attempt to 
receive as many bids as possible, is simply impractical, and ultimately does not result in 
additional cost savings. While the Competition in Contracting Act (CICA) also includes 
provisions for exemption, its total focus on competing contracts continues to influence 
the decisions of contracting officers today. 

Interestingly, the CICA also formali z ed the concept of best value, introducing the 
tradeoff method in the process. A focus on keeping costs low led to the predominant use 
of sealed bidding to acquire government products and services (Lohfeld, 2015). The 
CICA’s importance allowed contracting officers to consider different evaluation factors 
such as management or past performance in addition to price, even allowing the 
government to weigh them as significantly more important than price. 

5. Defense Acquisition Workforce Improvement Act of 1990 

Despite the implementation of the CICA, examples of unreasonably high 

acquisitions persisted, with key inquiries led by the Packard Commission (Gates et ah, 

2008). Additionally, the growing amount of legislation from Congress and regulation 

from each of the component services required a dedicated acquisition workforce with 
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extensive training on mandatory policies. In 1990, Congress passed the Defense 
Acquisition Workforce Improvement Act (DAWIA), which (1) formed the Defense 
Acquisition University (DAU), (2) directed the services to create formal acquisition 
career paths, and (3) led to the implementation of several DOD instructions that continue 
to tailor the acquisition workforce (Gates et ah, 2008). While previous legislation focused 
on contracting policy, the DAWIA was drafted specifically to shape and monitor the 
skills and capabilities of acquisition personnel. 

6. Federal Acquisition Streamlining Act of 1994 

A mere four years after the implementation of the DAWIA, the federal 
government again decided to alter the nature of defense acquisition in a sweeping way. 
During the early 1990s, the Clinton administration led a major effort to review and 
respond to the cost of federal procurement, which was growing at an alarming rate 
(Barry, 1995). The research concluded that government-specific regulations made up a 
significant portion of contract costs; the TINA alone “add[ed] about a 20 percent 
premium to government contracts” (Barry, 1995, p. 124). For some firms, this complex 
web of regulations was simply not worth the effort, and they declined to do business with 
the government altogether (Barry, 1995). 

In response, the Federal Acquisition Streamlining Act (FASA) aimed to harness 
the efficiency of the private sector by shifting to commercial specifications and relying 
on industry standards and market forces to ensure the best prices. To this end, the law 
created a preference for commercial products, established electronic posting of 
requirements, and allowed additional exemptions for contingency contracting—especially 
relevant after lessons learned during the first Gulf War (Barry, 1995). Additionally, the 
FASA mandated awards to small business concerns for purchases below the simplified 
acquisition threshold, but above the newly created micro-purchase threshold. Finally, the 
FASA created what we now know as simplified acquisition procedures, through a 
complete rewrite of FAR Part 13 and the repeal of 225 procurement statutes (Barry, 
1995). 
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7. Clinger-Cohen Act of 1996 

Originally passed as Divisions D and E of the 1996 National Defense 
Authorization Act, the Clinger-Cohen Act (CCA) included fixes to several FASA 
provisions, as well as enacted new policies (Sherman, 1997). Division D, originally 
known as the Information Technology Management Reform Act (ITMRA), 
fundamentally changed the way the DOD acquired information technology (IT) services 
by removing the General Services Administration’s (GSA’s) role as manager of IT 
acquisitions. Additionally, the ITMRA directed each service to create a chief information 
officer (CIO) position to consolidate and report on IT issues. 

Division E, also known as the Federal Acquisition Reform Act (FARA), included 
several significant changes as well. The FARA refined the concept of competitive range 
by allowing contracting officers “to limit the size of the competitive range to the greatest 
number of proposals that will permit efficient competition” (Rumbaugh, 2010, p. 24). 
This update considers the timeline of the contract management process in addition to 
transparency concerns; acquisition teams now have the flexibility to determine the 
appropriate priorities for each procurement. The FARA also introduced FAR 13.5, which 
was referred to as the Test Program for Commercial Items, and has recently become a 
permanent addition to the FAR (Sherman, 1997). One interesting note about the FARA is 
the timeline of its enactment—agency officials had not yet finished implementing the 
Federal Acquisition Streamlining Act (FASA). One analyst dubbed the entire process 
“Reforming [the] Reform” (Sherman, 1997, p. 1), as the flurry of new legislation 
outpaced the agency’s ability to execute them. 

8. Weapon Systems Acquisition Reform Act of 2009 

Just over a decade after the Clinger-Cohen Act, Congress passed the Weapon 
Systems Acquisition Reform Act (WSARA), which deals specifically with Major 
Defense Acquisition Programs (MDAPs). With the overall goal of reducing cost and 
schedule overruns—estimated at $3.1 billion and 22 months per program—it also made 
improvements to contract performance monitoring (Berteau, Hofbauer & Sanok, 2010). 
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To this end, the WSARA created four new oversight positions, added more stringent 
criteria for advancing milestones, and supplemented reporting requirements to Congress. 

As the record of legislation shows, the DOD has been tweaking acquisition 
procedures for decades, with no end in sight. While each change represents an 
incremental improvement in the acquisition process, regulators and industry players agree 
that more work remains. Of note, each piece of legislation ultimately rebalanced the four 
principles of the federal acquisition system. 

9. Better Buying Power 

The most recent piece of acquisition reform, however, did not originate from 
Congress. The creation of Secretary of Defense Ashton Carter and USD(AT&L) Frank 
Kendall, Better Buying Power (BBP) was originally a re-emphasis of existing policies, 
which ultimately grew into a unique piece of reform (Kendall, 2014). When asked. Carter 
explained that the concept behind the improvements was always an iterative process; no 
one initiative can succeed in a vacuum (Kendall, 2014). Now in its third installment, the 
program lives on, building upon the same core principles that started five years ago 
(Kendall, 2010). One of those principles is the concept of affordability. The first version 
of the initiative stressed the importance of “targeting affordability and controlling cost 
growth” (Kendall, 2010), while the update two years later highlighted the need to 
“mandate affordability as a requirement” (Kendall, 2012). In the latest update, the focus 
has narrowed to “enforce affordability caps” (Kendall, 2014), but the emphasis on 
controlling costs is still present. 

However, the importance of affordability cannot come at the expense of other 
acquisition objectives. As reformers are quick to point out, the goal is process 
improvement, not to “cut a percentage off the contractor’s bid” (Hagen, 2014). Likewise, 
cheaper programs cannot come at the expense of warfighter capability. In fact. Carter and 
Kendall designed the original set of BBP initiatives around the opposite premise: the 
ultimate goal is to direct savings into operational accounts (Corin, 2013). Instead of 
buying fewer or cheaper items, the architects of BBP understood the need to buy smarter. 
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But what does buying smarter look like? In order to answer this question, it is 
important to understand how the government currently acquires goods and services. The 
following is a brief discussion of the contract management process. 

D. GOVERNMENT CONTRACT MANAGEMENT PROCESS 

Regardless of the good or service, government contracting typically follows the 
same timeline of six basic steps (Garrett, 2007): 

• procurement planning 

• solicitation planning 

• solicitation 

• source selection 

• contract administration 

• contract closeout 

Each of these steps represents a broad overview of fundamental functions, and 
can be as detailed or a brief as necessary. For example, an order for common printer toner 
certainly requires less contract administration than a multi-million-dollar construction 
effort. Nevertheless, each of the above steps is present in virtually all government 
acquisitions, even if only briefly. 

1. Procurement Planning 

The first step in the contract management process is procurement planning. 
During this step, the buyer contemplates whether contracting is even necessary, often 
categorized as a make-or-buy decision (Garrett, 2007). While the government seldom 
produces physical goods organically, uniformed personnel are capable of performing 
many federal services. In these cases, analysts conduct internal cost-benefit analyses to 
determine the best approach to fulfill the need. Additionally, this step typically includes 
market research to define the need itself, as well as to understand the capabilities of 
potential bidders (Garrett, 2007). The implication of this step should not be understated. 
Procurement planning sets the foundation for three key decisions: contract type (fixed- 
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price versus cost-reimbursement), procurement method (sealed bidding versus contract 
by negotiation), and source selection method (LPTA versus tradeoff). 

2. Solicitation Planning 

Once the acquisition team decides to proceed with a contract, solicitation planning 
begins. During this step, the government documents the details of the contract, from 
evaluation criteria to length of the relationship with the contract winner (Garrett, 2007). 
While government officials must operate within the confines of procurement regulations 
throughout the process, policymakers have developed various acquisition techniques as 
part of the ongoing reform previously mentioned. Officials have introduced each new 
strategy in order to assist acquisition teams in procuring the right item in a fair and 
transparent manner. We discuss these in detail later; however, it is important to note that 
like procurement planning, this step has a significant impact on the success of the 
contract. 


3. Solicitation 

Solicitation formally begins once the government publicizes the request for goods 
or services. As firms review solicitation documents and present offers to the government, 
officials must be prepared to answer questions, provide additional information on the 
requirement, and amend documents as necessary. One important note concerning 
solicitation is the efficiencies gained by recent technological developments. For example, 
the introduction of FedBizOpps, the online government point of entry, allows contracting 
offices to publicize notices instantly from a computer, rather than mailing out dozens of 
packages to prospective bidders. 

While a poorly conducted solicitation can negatively affect the contract 
management process, successful completion of this step hinges on the framework 
established during solicitation planning. 

4. Source Selection 

Source selection consists of applying the evaluation criteria stated in the 


solicitation to the offers received. Like the previous step, source selection requires an 
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adherence to the overall strategy selected in the initial phases of the contract management 
process. Depending on the size and complexity of the procurement, this may involve 
source selection boards, technical panels, and any other expert required to evaluate the 
offers received (Garrett, 2007). Further, the source selection may involve directly 
negotiating with one or more vendors on price, technical factors, or personnel. Finally, 
the acquisition team selects the winner during this step; it is the most vulnerable to 
protests from unsuccessful vendors. 

Due to the visibility of source selections, senior officials have placed a specific 
emphasis on the step, with many of the previously mentioned reform efforts centered on 
source selection. However, source selection is actually an execution of the evaluation 
strategy. By the time the government has advanced to source selection, it is typically too 
late to make significant changes without regressing back to a previous step. Therefore, it 
is critical for acquisition teams to adequately perform the initial planning activities. 

5. Contract Administration 

Contract administration, like all other steps, is vital to the success of the contract 
management process. Moreover, the length of contract administration often dwarfs each 
of the other steps, particularly in multi-year contracts. Successful contract administration 
consists of adhering to procurement policy and enforcing the contract as written; often 
complications occur when one or both of the parties fail to follow the stated agreement. 
While change requests and modifications can occur in any contract, the key to effective 
contract administration is a clear contract that stipulates the expectations of each party. 

6. Contract Closeout 

The final step in the contract management process is contract closeout. This step 

includes confirming that all required work has been accomplished, as well as any other 

details related to the contract. On some occasions, performance is terminated early; this 

can occur due to either mutual agreement or a breach of contract by one of the parties 

(Garrett, 2007). Clearly, the preferred outcome is successful fulfillment of obligations by 

both the buyer and the seller. However, when complications do arise, they are often 

caused by decisions made before award. In fact, despite the abundance of guidance 
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concerning source selection and post-award actions, solicitation planning remains “one of 
the most critical functions in contract management” (Garrett, 2007, p. 24). In order to 
understand the significance of this step, one must recognize the choices available to 
acquisition teams before contract award. 

E. PRE-AWARD ACQUISITION DECISIONS 

One useful analogy for describing the various alternatives available to contracting 
officers is a decision tree. While dozens of contract types, procurement methods, and 
source selection methods are available, decisions made early in the process narrow the 
field of future possibilities. For example, if an acquisition team determines a product or 
service to be other than commercial, the procedures offered under FAR 12 are no viable 
procedural options (FAR 12.102(a)). 

I. Contract Type 

One of the first decisions available to acquisition teams is the type of contract 
used to procure a good or service. Covered in FAR 16, contract type influences not only 
the procurement method and source selection method, but the entire contract management 
process as well. 

Fixed-price contracts, the first type, are characterized by generally firm prices. 
The total amount paid to the contractor is typically only adjusted to reflect incentive fees, 
economic adjustments concerning component pricing, or the use of a contract clause to 
cover an unforeseen situation (FAR 16.201). Since the contract price is generally 
determined before performance begins, the contractor shoulders most of the risk. If the 
effort costs more than originally planned, the contractor typically earns less profit, while 
the government may face little to no consequences. 

In response, contractors typically factor this risk into their proposals, building in 
larger profit premiums for work that is complex or prone to unknown hazards. For this 
reason, fixed-price contracts are recommended for clearly defined goods and services, 
such as commercial-off-the-shelf items, or efforts that a contractor has performed in the 
past (FAR 16.104(1)). 
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In situations where the required work is complex, undefined, or otherwise 
difficult to estimate initially, acquisition teams may use the second type, cost- 
reimbursement contracts. Cost-reimbursement contracts differ from fixed-price contracts 
in that the government reimburses the contractor for all allowable, reasonable, and 
allocable costs incurred in performance of the contract (FAR 16.301-1). Since the 
government assumes significantly more risk under cost-reimbursement contracts, the 
profit premium afforded to contractors is typically lower under this arrangement (DFARS 
215.404-71-3(c)). 

While fixed-priced contracts are generally recommended for use whenever 
practical, cost-reimbursement contracts are only allowed after an extensive review of the 
requirement, the proposed contractor, and the contract administration team (FAR 16.301- 
3(a). Additionally, acquisition teams are prohibited from using cost-reimbursement 
contracts to procure commercial items, limiting their use to complex, uniquely 
governmental efforts. 

2. Procurement Method 

After selecting the appropriate contract type, acquisition teams must determine 
the best procurement method for the acquisition. The two basic options differ primarily in 
the evaluation of proposals, yet the consequences of this decision are reflected in nearly 
every step of the contract management process. 

Sealed bidding, the first method, consists of receiving vendor bids and selecting a 
contractor chiefly based on price, or price-related factors (Rosa da Silva, 2013). It is 
relatively simple, as contracting officers are simply responsible for determining the 
lowest bid price that conforms to the details of the solicitation. Sealed bidding has a 
prominent role in the history of federal contracting; until World War II, it was the 
preferred option for acquiring supplies and services (Rosa da Silva, 2013). However, 
sealed bidding relies almost entirely on price-related criteria, and severely limits 
discretion in choosing a contract type; only firm-fixed-price contracts and fixed-price 
economic price adjustment contracts are permitted (FAR 14.104). As sealed bidding only 
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allows a narrow range of acquisition decisions, it is not suitable for many of the complex 
good and services contracting officers are responsible for buying today. 

The second method available is contracting by negotiation. This alternative was 
first introduced as part of the CICA under the broad concept of best value, and allows 
government officials to discuss proposal details with offerors after receipt of proposals. 
Contracting officers can negotiate in competitive procurements to determine the relative 
merits of various proposals; or, in noncompetitive instances, to discuss pricing (Rosa da 
Silva, 2013). Based on the wide discretion government officials have under negotiation 
procedures, each acquisition must have clearly stated evaluation criteria so vendors know 
how to shape their proposals. This range of available evaluation options is known as the 
best value continuum, and is explained in the following section. 

3. Source Selection Method 

In order to understand the differences in source selection methods, one must first 
review the best value continuum. Specifically, it is important to understand the two 
extremes of the continuum to assess the two conflicting goals of government acquisition. 
The first goal of contracting officers is to provide maximum quality to the end user. 
Delivery time, superior customer service, and innovative technical approaches are all 
examples of quality. Regardless of the contract type, procurement method, or source 
selection method selected by the acquisition team, the contract will always express some 
level of quality requirements in a contract, because an acquisition for insufficient goods 
or services is never in the best interest of the government. On the best value continuum, 
selecting the highest technically rated offeror represents a total commitment to quality 
(FAR 15.101-l(a)). 

However, as part of instituting contract by negotiation, the CICA also placed 
restrictions on the method. Chiefly, the act requires federal officials to consider price in 
every acquisition, regardless of where it falls on the best value continuum (Rumbaugh, 
2010). While acquisition teams have the discretion to weigh price as more or less 
important than other factors, the mandatory inclusion of price essentially precludes 
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evaluation criteria designed specifically to select the highest technically rated offer 
without regard to cost. 

The second objective inherent in the federal contract management process is the 
need to ensure the efficient use of taxpayer dollars. Similar to quality, stewardship of 
public funds is required for all contracts, and is stated in the first guiding principle for the 
federal acquisition system. Maximum attention to price as an evaluation factor is known 
as LPTA, and it serves as the opposite extreme of the best value continuum. 

Since an evaluation strategy aimed purely at selecting the highest technically 
rated offer without regard to cost is prohibited by statute, the full best value continuum is 
technically not available to acquisition teams. Instead, a modified version of the spectrum 
remains in its place, with two main options: LPTA and tradeoffs. The following is a brief 
explanation of each method, along with relative strengths and weaknesses. 

F. LOWEST PRICE TECHNICALLY ACCEPTABLE 

Under LPTA source selections, the government first ranks all offers in reverse 
price order, and then reviews them in that order until a technically acceptable offer is 
identified. If an offer is found not to be technically acceptable, it is eliminated, and the 
government proceeds to the next lowest priced offer. Under this basic framework, 
acquisition teams are not authorized to accept a proposal other than the technically 
acceptable low offer (FAR, 15.101-2). 

Unsurprisingly, the FAR recommends LPTA acquisitions when “best value is 
expected to result from selection of the technically acceptable proposal with the lowest 
evaluated price” (FAR 15.101-2(a)). More specifically, LPTA procurements are preferred 
when the requirement is well-defined and when the government does not anticipate a 
significant risk of contractor default (GAO, 2014). Since acquisition teams do not reward 
vendors for proposing technical approaches that exceed the minimum requirements, the 
strategy is recommended for requirements that will not require an innovative solution. 
Put another way, LPTA contracts are ideal for what private defense firm The Analytical 
Services Corporation (TASC) describes as the “acquisition of commercial or non¬ 
complex services that are clearly defined and expected to be low risk” (TASC, 2012, p. 
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2). For example, the GAO highlights the government’s high usage of LPTA source 
selections in commercial acquisitions of fuel (2014). Even for large contracts, fuel is a 
relatively well-understood requirement for buyers and sellers, and as long as a vendor can 
meet the basic contract terms, there is no need to review quality factors among the offers. 

As a source selection method, LPTA satisfies many of the federal acquisition 
guiding principles. As contracting officers are required to accept the lowest price offer 
that meets the government need, the customer’s need is expressly satisfied in terms of 
cost, while timeliness and performance are ostensibly included as well. Additionally, 
because LPTA acquisitions tend to be simpler than tradeoffs, contracting offices can 
move more quickly through the six-step contract management process, reducing 
administrative operating costs. Linally, the generally inflexible nature of the LPTA 
source selection method does not grant contracting officers discretion, which serves as a 
guard against the appearance of favoritism, promoting the perception of integrity, 
fairness, and openness. 

G. TRADEOFF 

By contrast, tradeoff acquisitions allow contracting officers the flexibility to 
weigh non-price factors as part of an overall evaluation assessment. Under this method, 
the acquisition teams have the flexibility to select other than the LPTA offer if the 
perceived benefits of a higher priced offer outweigh the additional cost (PAR 15.101- 
1(c)). This decision must conform to the evaluation criteria stated in the solicitation; 
however, the mere ability to weigh multiple components of an offer can be useful in 
complex, technical acquisitions. 

As the GAO (2014) notes, tradeoffs are valuable when the past performance or 
technical capacity of a contractor is important; goods or services that require a 
contractor’s experience throughout the acquisition process may warrant the use of the 
tradeoff method. These are most commonly found in complex acquisitions where the 
government anticipates unknown challenges in the future, that is, a specific need cannot 
be identified at the time of award, but the increased capability of qualified contractors 
may be beneficial. This dialogue of evaluation factors and their relative importance is 
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known as an integrated assessment, and is essentially “the crux of the entire source 
selection” (Slate, 2004, p. 38). 

Compared to the LPTA method, tradeoffs emphasize providing maximum 
performance for the customer, although costs may increase as a result. However, 
acquisition teams rely on price competition among offerors to ensure fair and reasonable 
offers (FAR 15.402(a)(2)). Additionally, proponents of tradeoffs argue that the initial 
costs of a higher priced vendor are ultimately more efficient, as the incentive structure 
encourages vendors to avoid cutting costs that could jeopardize the effort after award. By 
this logic, the resulting contract is less likely to require modifications after award due to 
the contractor’s increased capability. Finally, price competition is still typically present. 

In terms of fairness and openness, tradeoffs certainly expose the government to 
the perception of internal bias. While tradeoffs are specifically structured to reduce this 
risk—for example, acquisition teams are required to state the relative importance of price 
versus non-price factors—the strategy ultimately carries the perception of less 
transparency. 

H. IMPLICATIONS OF EACH SOURCE SELECTION METHOD 

While in some cases the choice between the LPTA and tradeoff source selection 
methods is clear, the majority of acquisitions require government teams to analyze the 
situation and weigh the relative benefits of each method against their costs. To 
understand this process, we have summarized the key implications of each method. 

However, it is important to note that these conclusions are not necessarily 
representative of the entire population of federal contracts. Many of the findings come 
from GAO reports, which, while informative, are typically based on small sample sizes 
(15 to 20 contracts). In other cases, the conclusions are drawn from academic journals or 
sentiments of acquisition personnel (GAO, 2014), which also lack the statistical 
significance of a data analysis. 


21 



1 . 


LPTA 


The most significant weakness inherent in LPTA procurements is the potential for 
unrealistic pricing. The reliance on price as the final discriminator in LPTA source 
selections has a negative tendency to encourage a race to the bottom by competing 
vendors. The Department of State’s (DOS’s) contractor, Armor Group North America 
(AGNA), provides an excellent case study for this behavior, as the contractor’s 
shortcomings were directly related to their unsustainable pricing structure (CWC, 2009). 
Rather than inexperienced management or unethical employees, the majority of 
contractor violations resulted from cost-cutting measures: lack of security vehicles, third 
country national personnel incapable of communicating in English, and shortage of 
employee uniforms (CWC, 2009). The AGNA example is particularly troubling because 
the DOS was not allowed to utilize tradeoffs for overseas security acquisitions at the 
time, requiring the use of an inappropriate source selection method. 

In order to combat this, acquisition teams often employ cost realism: a methodical 
review of proposed pricing against the required performance to identify and respond to 
unrealistically low offers. Cost realism was originally designed as a technique to mitigate 
unrealistic pricing on cost-reimbursement contracts, where the potential risk to the 
government is the greatest, but contracting officers can also use cost realism language in 
fixed price contracts in to achieve the same result (Lohfeld, 2014). However, this solution 
must be pre-meditated; evaluating offers for cost realism without specifically stating so in 
the solicitation represents a departure from the source selection criteria, and a “fertile area 
for procurement protest” (Lohfeld, 2014). 

Despite efforts to promote cost realism, the perceived threat to offerors of 
unrealistically low pricing by competitors is often enough to dissuade a bid, because the 
resources spent creating an offer appear to be a waste (CWC, 2009). Regardless of 
whether the government would have chosen these technically superior contractors, their 
presence increases the competitive nature of the acquisition, increasing government 
value. Removing them from the acquisition is not good for the government. 
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In other cases, the need to remain competitive may persuade firms to minimize 
their staff and research funding in order to reduce operating costs. As noted by TASC 
(2012), general and administrative (G&A) costs are typically the business expense 
eliminated, followed swiftly by research and development (R&D) funding. While large 
firms such as Lockheed Martin and Northrup Grumman typically have several projects to 
move personnel around, small firms may have to radically alter their entire operation to 
win bids, permanently reducing their potential technical innovation. 

Further, among contractors who maintain their original staffing levels, there is no 
incentive to present a high quality product or service (Nichols & Totman, 2013). In fact, 
LPTA procurements actually discourage any level of effort beyond the stated 
minimum—quoting additional resources simply drives up the proposed cost without 
increasing the attractiveness of an offer. In a recent National Contract Management 
Association (NCMA) presentation, the organization advised vendors to bid for the 
absolute minimum level of work required and to cut supervisory personnel on LPTA 
projects, even though the strategy “could be deemed risky to some companies” (Calhoon 
& Sybert, 2012). 

2. Tradeoffs 

Conversely, tradeoffs often suffer from the perception of increased costs. 
Ultimately, trading higher prices in exchange for a more robust technical proposal or 
superior past performance means paying some form of performance premium, as part of 
the integrated assessment process. While the acquisition team may be fully justified in 
paying a higher cost, outside observers—both public and private—may disagree. The 
GAO noted that, among best-value acquisitions, the percentage of tradeoffs declined 
steadily from 2009 to 2013, the period for which they analyzed the metric (2014). 
Additionally, DOD contracting officers and program managers echoed these findings, 
attributing the heightened use of LPTA to shrinking budgets caused by sequestration. 
From the industry perspective, this opinion is repeated; TASC has attributed the decline 
in tradeoffs to shrinking budgets (TASC, 2012). While the government is required to 
evaluate price reasonableness for all offers, preferably through the use competitive 
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proposals (FAR 15.305(a)(1)), the increased scrutiny brought on by the tradeoff method 
appears to influence source selection method nevertheless. 

Additionally, the increased flexibility offered by tradeoff requires a pool of 
adequately trained personnel to successfully conduct the source selection. Unlike LPTA 
source selection, which only assesses the threshold of acceptability, acquisition teams 
must define and quantify each evaluation factor as part of each procurement’s specific 
integrated assessment. The GAO noted that “developing non-cost factors that 
meaningfully discriminate between offers” can be particularly difficult (2010, p. 17). 
Their report also highlights the need to bring in personnel with specific expertise, citing a 
U.S. Army Corps of Engineers project where water flow experts were brought in to 
analyze the merits of different proposals (GAO, 2014). 

Moreover, the requisite training cannot be accomplished by a new DAU offering. 
In interviews with the GAO, DOD officials stressed the importance of on-the-job training 
for acquisition personnel, specifically when deciding between the two source selection 
methods, as well as justifying the decision to select a higher priced offer (GAO, 2014). 
This is particularly relevant today because the DOD increased the size of the contracting 
workforce in the years leading up to fiscal year 2015. Training these new personnel in the 
nuances of tradeoffs will likely take decades. 

Another consequence of the complicated nature of tradeoffs is the increased risk 
of protests from losing offerors. Within federal acquisition, two factors significantly 
increase the risk of protest: dollar value and agency discretion. As Carl Gebo, a partner at 
Adorno & Yoss (2009), explains. 

In negotiated procurements, the interaction between offerors and the agency 
can be far greater than in sealed bid procurements. In seeking clarifications, 
conducting discussions, and making presentations, suspicions of 
prejudicially different treatment thrive and add to the incentive to protest. 

The opportunity to influence those evaluations through those agency 
contacts is real and even part of the process. When looked at in retrospect, 
through the prism of a lost contract, offerors are inclined to believe that 
undue influence or unfair treatment occurred, (p. 30) 
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Based on the government’s increased use of the tradeoff method for high-dollar 
acquisitions, particularly above $25 million, as well as the subjective nature of 
conducting an integrated assessment, the risk of protest(s) is much higher for these 
procurements. 

Even among highly trained personnel who manage to avoid protests, tradeoffs can 
often be lengthy affairs, especially when compared to LPTA procurements. Developing 
meaningful criteria, effectively stating the evaluation methodology, and conducting the 
integrated assessment are all deliberate processes, resulting in longer procurements. 
During a GAO study, acquisition personnel echoed the measured pace of the tradeoff 
method, citing a typical timeline of 18 to 24 months for completion (2010). While the 
GAO based its findings on individual interviews rather than contract data collection, the 
point remains. This sentiment is echoed by industry, which discredits the LPTA method 
but grudgingly admits its streamlined efficiency (Lohfeld, 2015). 

The final, and perhaps most troubling, drawback of the tradeoff method is the 
government’s apparent lack of actually trading off price and non-price factors. In an 
FY2009 study, the GAO found that DOD contracting officers utilized the tradeoff 
method in 68% of their sample (2010). However, of these, the government selected a 
higher priced offer less than a quarter of the time. Even more troubling, selecting the non- 
low offer resulted in a protest rate of 72%, with nearly a third of protests resulting in 
either acquisition cancellation or changed award decision (GAO, 2010). Put simply, the 
government seems unwilling or unable to utilize the principal benefit of the tradeoff 
method. Given the increased time and training required, as well the increased risk of 
protests, the illusion of choice may not be in the government’s best interest. 

I. SUMMARY 

In this chapter, we have reviewed the context for the tradeoff source selection 
method. Prior to the CICA, the government typically utilized some version of LPTA, 
with a primary focus on acquiring goods and services at the lowest price. As government 
regulation evolved, so did the guiding principles of acquisition, resulting in objectives 
other than cost savings. The federal government introduced tradeoffs along with the best 
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value continuum as a way to encourage innovative solutions for complex procurement 
problems. Based on the relative strengths and weaknesses of the tradeoff method, it is 
typically advised for complex, developmental, less-defined procurements. 

However, as discussed previously, many of these conclusions are drawn from 
incredibly small sample sizes, while others simply express personal experiences. In the 
next chapter, we explain the methodology of our study in order to test the validity of 
these claims using contract data. 
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III. METHODOLOGY 


A. INTRODUCTION 

In this section, we examine our process for collecting and analyzing relevant data 
to answer the research questions of this study. We hope that our approach can pave the 
way for future studies of this subject across all military branches. This section will 
provide a brief examination of the contracting structure and systems used in the Air 
Force. The subsequent sections explain the data collection method and the unique 
elements of our research. 

B. CONTRACTING STRUCTURE AND SYSTEMS 

This section provides an overview of the structure and systems used in the Air 
Force to carry out the contracting mission. 

1. Base Operational Contracting 

Within the Air Force, most installations include a contracting contingent 
(typically a fully staffed squadron) responsible for procurement needs across the base. 
Contracting officers (COs) and contracting specialists within the squadron are responsible 
for procurement planning, solicitation planning, solicitation, source selection, contract 
administration, and contract closeout for every acquisition (Garret, 2007). The discharge 
of these duties involves ensuring documentation of such actions is complete within the 
contract file and all associated databases. 

2. Federal Procurement Data System-Next Generation 

The Federal Procurement Data System-Next Generation (FPDS-NG) provides the 
platform for COs to report contract data to government officials. Within this system, each 
contract action has a corresponding Contract Action Report (CAR) that captures a 
significant amount of data relating to the contract, including dollar amount, length of 
performance, number of offers received, and the extent to which the purchase was 
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competed. These data are unique to each contract action and are submitted and 
maintained in the FPDS-NG system. 

3. Federal Business Opportunities 

Another valuable source of contract information is the government-wide point of 
entry, otherwise known as the Federal Business Opportunities (FBO) website 
(www.fbo.gov). Contracting squadrons upload solicitation documents to the website to 
inform potential contractors of government requirements. The solicitation, maintained on 
the FBO website, contains proposal preparation and submission instructions, states the 
evaluation factors that the source selection team will use to assess each proposal, and 
identifies which source selection method will be used to make the final award decision. 
The FBO database stores solicitations and all related amendments, as well as additional 
information associated with the requirement (e.g., drawings or specifications, answers to 
questions posed by potential vendors, etc.). 

4. Contractor Performance Assessment Reporting System 

The Air Force and other government agencies use the Contractor Performance 
Assessment Reporting System (CPARS) to report their periodic evaluations of contractor 
adherence to contract terms. This system stores contractor performance evaluations and 
supporting rationale for each contract. FAR Subpart 42.15 provides instructions for 
evaluating contractors based on the elements of cost control, technical, schedule, business 
relations, and subcontracting. Within these broad categories, the contractor is given an 
evaluation according to a Likert-style scale, which is discussed further in the next 
chapter. As demonstrated in the following sections and chapters, these ratings serve as a 
proxy rating of overall contract performance; higher ratings indicate compliance with 
contract requirements and positive performance. 

C. DATA COLLECTION 

Although the aforementioned systems contain data relevant to our research, we 
needed access to more in-depth data that is not available in any system or database. As 
such, we traveled to two Air Force contracting squadrons to review their contract files 
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and extract the necessary data. Specifically, we captured contract outcome data from 
contracts that used the tradeoff source selection method and compared that information 
against the same data points gathered from contracts using the LPTA source selection 
method. 

We focused our research on contracts awarded by the Air Force’s 21st 
Contracting Squadron located at Peterson Air Force Base, and the 50th Contracting 
Squadron located at Schriever Air Force Base. We felt that the Colorado Springs area, 
where these bases are located, offered significant potential for gathering large amounts of 
data, due to the close proximity of the two bases. Also, Peterson Air Force Base is the 
headquarters of Air Force Space Command, and we knew that it would provide a 
sufficient mix of contracts likely to involve both tradeoff and LPTA procedures. With the 
help of the Air Force Installation Contracting Agency (AFICA), we contacted both bases 
to request that they identify contracts with the following criteria: 

• available CPARS data 

• high dollar amount, preferably valued above $700,000 

• an equal number of contracts that utilized tradeoff and LPTA source 
selection methods 

By coordinating with each point of contact, a sufficient number of contracts were 
made available to us upon our arrival at each base. Between the two installations, we 
accessed data from 33 contract files dealing with a broad range of requirements. 

The five categories of data that we were interested in locating for our study were: 

• basic contract information 

• acquisition complexity 

• environmental factors 

• outcome variables 

• other data (relating to discussions) 

Basic contract information included the main identifying features of a contract, 

such as the contract number and description, as well as the buying organization 

(squadron). The North American Industry Classification System (NAICS) code and 
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Product Service Code (PSC) were also captured as further identifying characteristics for 
each contract. 

Acquisition complexity was measured using such factors as contract type, set- 
asides used, length of contract, number of contract line items, and dollar value. In this 
section, we note the source selection method utilized, as well as the number and 
description of evaluation factors and subfactors. Industry involvement was measured by 
whether or not conferences were held in relation to solicitation and communication of 
requirements, how many offerors submitted proposals, and whether there were multiple 
awards or a single contract award. Incentive and award fee criteria, where present, were 
also relevant to the acquisition complexity measure. 

Environmental factors included the number of reviews, as well as the utilization 
of Multi-Functional Independent Review Team support. Here we also gave each contract 
file a rating for level of documentation on a five-point scale where l=well below average, 
2=below average, 3=average, 4=above average, and 5=well above average. We also 
included an assessment of the state of the U.S. economy at the time of contract award, in 
order to assess the larger economic context of each acquisition. 

Outcome variables were crucial to our study because they provided the dependent 
variables and basis for comparison of the two source selection methods. The PALT and 
CPARS ratings (further expounded upon in the next chapter) were the main factors used 
to evaluate contract outcomes. Other important measures of contract outcome included 
protest data. Earned Value Management (EVM) data, number of solicitation 
amendments, and earned award fee. 

The last category of information, other data, included discussions between the 
government and vendors during source elections. Eor example, we looked at requests 
from the contracting office to offerors for clarifications of their proposals. When the 
source selection team notices a deficiency in an offeror’s proposal, a deficiency report or 
an Evaluation Notice (EN) is issued to make the problem known. We counted the number 
of ENs, as well as the number of rounds of ENs, to better understand the quality of 
proposals as well as the effectiveness of the source selection. The final data point we 
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collected was the basis for award; if the contract was awarded on any basis other than 
cost or price, we noted the determining factor. 

D. UNIQUE ELEMENTS OF OUR RESEARCH METHOD 

By working with our contacts at each base prior to our travel, we ensured that 
there would be a sufficient and equal number of contracts that utilized the two different 
source selection methods. If we had accessed a list of each base’s contracts and requested 
specific contracts be made available, we might have encountered the problem of having 
too many sample cases of one source selection method and not enough of the other. Our 
contacts’ prior knowledge of each contract helped us avoid that problem. By requesting 
their assistance, we also avoided the potential pitfall of pre-selecting contracts for 
evaluation only to find upon arrival at the base that those contracts had been closed and 
staged. 

Another positive element of this approach, albeit intangible, was that it involved 
other contracting specialists and contracting officers who would otherwise not have taken 
part in the study. Sharing responsibility for this research helped to disseminate the 
thought process and goals of this research down to the squadron level. By involving 
others, they are now aware that AFICA is engaged in research that may streamline the 
procurement process and improve contracting operations. As contracting personnel 
communicate and transfer across squadrons, other installations will have an increased 
awareness of the data needed for future studies associated with this research. 

One unfortunate finding was the lack of EVM data. Major defense systems 
acquisition programs would be more likely to employ EVM practices. Although Peterson 
AEB and Schriever AEB both managed high dollar contracts dealing with space systems, 
they dealt mainly with system support rather than system development; therefore, the 
level of complexity did not necessitate EVM processes. Thus, we were unable to capture 
EVM data from the contracts made available to us. 

Another challenge of our research method was the sheer magnitude of the contract 
files to be examined. Some files were unexpectedly large, with portions stored in 
different locations within the office building. Several times, a member of our group 
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would begin examining the pre-award data for a contract only to subsequently discover 
that the award document was stored elsewhere. Each installation’s security measures 
were unique, and this caused some delays in finding crucial documents. This was 
particularly the case at Schriever AFB. While this issue did not significantly impede our 
progress, it did slow the process and constrain our already limited time on site. 

E. SUMMARY 

Overall, our data collection methodology accomplished the end goal of gathering 
a sufficient amount of data to examine our research questions. Colorado Springs provided 
an excellent location for our research; we were able to examine the contracts of two 
separate squadrons, which provided us with a mix of LPT A and tradeoff contracts and 
their corresponding outcomes. The next chapter presents the results of our data analysis. 
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IV. RESULTS AND RECOMMENDATIONS 


A. INTRODUCTION 

Once we collected the data mentioned in the previous chapter, we used statistical 
software to perform a series of analyses. In the following sections, we discuss the 
assumptions tested before we ran the program, followed by the results of the data analysis 
and conclusions to be drawn therefrom. 

B. DATA DESCRIPTION 

For this analysis, we used two main outcomes, or dependent variables (DVs): 
procurement administrative lead time (PALT) and Contractor Performance Assessment 
Reporting System (CPARS) data. PALT assesses time to contract by calculating the 
number of days between receipt of the requisition and contract award. It is a continuous 
variable. CPARS serves as a proxy measure of contract success or failure by using the 
ratings given to each completed contract. CPARS ratings are given in Likert-style 
responses where l=unsatisfactory, 2=marginal, 3=satisfactory, 4=very good, and 
5=excellent. In this case, we used only combined CPARS scores as the second dependent 
variable. Specifically, we searched for differences in CPARS ratings between LPTA and 
tradeoff source selections by examining the average CPARS rating for each contract. The 
average rating was composed of the most frequently recorded CPARS scores: CPARS 
quality ratings, CPARS schedule ratings, and CPARS business relationship ratings. 

Our model has one independent variable (IV). Independent variables are those 
that can be manipulated by the researcher (or user) to evoke a change in the outcome, or 
DV. In this case, the IV concerns the source selection methodology used for the contract: 
LPTA or tradeoff. Source selection methodology is a choice made by the “user” (the 
integrated product team, which includes the contracting officer); hence it is considered an 
IV. Our IV is labeled LPTATO and it is a binary variable where 0=LPTA and l=tradeoff. 

Finally, there are three covariate variables. Covariates are secondary variables that 

can also affect the relationship of primary interest: the relationship between the IV and 

the DV. Specifically, covariates are variables other than the independent variable that 
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potentially affect the outcome variable, or DV. In this case, the covariates are (1) number 
of evaluation factors (NUMEVALFACTORS), (2) number of reviews (NUMREVIEWS), 
and (3) number of offers (NUMOFFERS). All are continuous variables. Theoretically, 
each of these covariates could affect the PAET positively (e.g., a larger number of any of 
the covariates could increase the PAET, or time to contract). In this case, we hope to 
parcel out the effects of the covariates in order to more clearly see the effect the 
contracting methodology (EPTATO) has on the outcome variables (PAET and CPARS 
ratings). It is worth noting that we wanted to use contract dollar value (VAEUE) as a 
covariate, however VAEUE did not pass the homogeneity of regression test (see 
paragraph 4 below), therefore we could not use it in the analysis. 

1. Descriptive Statistics 

Basic descriptive statistics for each variable are shown in Table 1. The table 
presents three figures for each variable: (1) the total for all the data, (2) the total for 
EPTA contracts, and (3) the total for tradeoff contracts. 

2. Data Limitations 

The data point NUMOFFERS, which proved to be an important variable, was 
collected from the Contract Action Report (CAR) on the Federal Procurement Data 
System—Next Generation (FPDS-NG). Taking this number from the CAR was more 
efficient and practical than counting each offer by hand; counting each offer ourselves 
would have proven to be an unwise use of our limited time available for data collection. 
The limitation with relying on the CAR, however, is that we became subject to human 
error that could have occurred at the time of initial entry into FPDS-NG. Information 
found in the CAR is only as good as the care taken to input the data. Using the CAR to 
count NUMOFFERS was the best approach; however, possible human error presents a 
limitation. 

The more substantial limitation of our data is its sample size and distribution. 

With only 33 cases, the sample size is small to moderate; however, the distribution of 

cases by source selection method is unbalanced. Power calculations suggest the need for 

14 cases for each method (i.e., 14 EPTA cases and 14 tradeoff cases) in order to achieve 
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adequate power (a = .05, (B = .80). The data are unbalanced with respect to the number of 
cases for each contracting methodology. There are 10 LPTA cases and 23 tradeoff cases. 
This unbalanced design can cause ambiguity about the mean as the intercept and make 
assignment of sums of squares more difficult. There are, however, solutions to these 
issues. A weighted mean can be used in place of the grand meani and the STATA 
software automatically handles the assignment of the sums of squares. Thus, we 
proceeded with our analysis despite these limitations. 


Table 1. Descriptive Statistics 


Variable 

Obs 

Mean 

StdDev 

Min 

Max 

PALT 

(days) 

33 

352.7879 

334.3218 

21 

1019 

10 

216.4 

198.8524 

30 

615 

23 

412.087 

366.4262 

21 

1019 

Average 

CPARS 

Rating 

22 

4.256061 

.7717661 

3 

5 

6 

4.302778 

.7895439 

3 

5 

16 

4.238542 

.7904925 

3 

5 

Number of 
Evaluation 
Factors 

29 

2.344828 

.6695341 

1 

4 

10 

2 

.4714045 

1 

3 

19 

2.526316 

.6966923 

2 

4 

Number of 
Reviews 

26 

9 

6.734983 

3 

28 

10 

9.6 

6.328068 

3 

25 

16 

8.625 

7.154253 

3 

28 

Number of 
Offers 

33 

5.666667 

5.677074 

2 

22 

10 

3.2 

1.549193 

2 

6 

23 

6.73913 

6.475249 

2 

22 


Bold = total for all data, non-italicized = LPTA, italicized = tradeoff 


3. Analysis 

Because our intent is to analyze differences in contract outcomes based on source 
selection methods, a group comparison statistical methodology is necessary. In other 
words, the source selection methods are divided into two groups, LPTA and tradeoff, and 
we seek to find if there are differences in contract outcomes between each group. 


^ The grand mean would be the intercept in a balanced design. 
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We used a technique called multivariate analysis of covariance, or MANCOVA, 
to assess group differences. MANCOVA essentially creates a new dependent variable 
using the information from the given DVs (PALT and CPARS ratings). This new 
dependent variable is created in a way that maximizes differences between the grouping 
variable (LPTA or tradeoff source selections). MANCOVA addresses the following 
questions: Are mean differences among the groups on a combination of DVs (after 
adjusting for covariate effects) likely to have occurred by chance? Taken from another 
angle, is there a significant difference between the mean value for PALT and CPARS 
ratings in the LPTA acquisitions versus the mean value for PALT and CPARS ratings in 
the tradeoff acquisitions, once the effect of the covariates (NUMEVALFACTORS, 
NUMREVIEWS, NUMOFFERS) have been parceled out? MANCOVA examines the 
relationships between the DVs (PAET and CPARS ratings) and the independent variable 
(choice of EPTA or tradeoff methodology) while taking into account the effect the 
covariates might have on the DVs. 

If differences in outcomes are found using MANCOVA, researchers typically 
choose to dig deeper into the differences using a univariate technique called analysis of 
covariance (ANCOVA). ANCOVA essentially performs the same “group differences” 
analysis as a MANCOVA; however, because it is univariate in nature, this method 
assesses one DV at a time (PAET or CPARS, rather than PAET and CPARS). This 
method helps researchers isolate where the difference is occurring (which IVs or 
covariates are affecting which DVs). 

4. Assumption Testing 

Before conducting the MANCOVA, certain assumptions about the data were 
tested. First, we assessed multivariate normality by examining density graphs, 
determining multivariate skewness and kurtosis, and examining the Doornik-Hansen test 
for multivariate normality (Doornik & Hansen, 2008). For the DVs, both PAET and 
CPARS were deemed to be non-normal. Both variables were normali z ed via a 
logarithmic transformation. All of the covariates (NUMEVAEF ACTORS, 
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NUMREVIEWS, NUMOEFERS) also required a logarithmic transformation, after which 
they were deemed to be multivariate normal. 

Second, we searched for outliers using Mahalanobis’ distance, and found no 
significantly influential outliers in our data. Third, we assessed linearity by examining 
scatter plots of (1) the paired DVs, (2) all pairs of covariates, and (3) all pairs of DV- 
covariate combinations for each source selection method (EPTA and tradeoff, a total of 
20 plots). The plots revealed that all relationships were linear in nature. 

Eourth, we assessed homogeneity of regression by performing an analysis of 
variance (ANOVA, which is practically the same as an ANCOVA but does not contain a 
covariate variable) that included the independent variable (EPTA or tradeoff), each of the 
covariates (NUMEVAEEACTORS, NUMREVIEWS, NUMOEEERS), and the 
interaction between the independent variable and the covariate. The interaction terms 
were not significant, which indicates that the relationship between the DVs (PAET and 
CPARS ratings) and the covariates is the same at both levels of the independent variable 
(EPTA or tradeoff). Hence, the assumption of homogeneity of regression is upheld. The 
covariate measuring the contract’s dollar value (VAEUE) failed this test; hence, it was 
dropped from further analysis. 

Eifth, we checked for multicollinearity by assessing the pooled within cell 
tolerance for each DV (PAET and CPARS). Multicollinearity is not an issue for the DVs 
when the independent grouping variable tradeoff is used; however, it may be an issue for 
the EPTA group. Eor the EPTA group, NUMOEEERS is very highly correlated with 
PAET (r = .94). This suggests that this covariate may be measuring many of the same 
characteristics that the PAET dependent variable is measuring. This can be troublesome 
because the high degree of similarity makes the variables redundant. To account for this 
issue, we chose to perform the MANCOVA twice: once with NUMOEEERS included 
and once with NUMOEEERS excluded. 

Einally, we checked for homogeneity of covariance matrices between groups (i.e., 
sphericity) using the multivariate test of means provided in STATA 12.1. This test checks 
whether or not population variances and covariances of both DVs (PAET and CPARS) 
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are equal for each of the independent variable groups (LPTA and tradeoff). The results 
showed that all grouping cells were homogenous. With all assumptions tested, we 
performed the MANCOVA. The results are provided in the following section. 

C. RESULTS 

With all the aforementioned assumptions tested, we performed the MANCOVA 
analysis. The following sections outline the results, first with NUMOFFERS included, 
second with NUMOFFERS excluded. 

1. Results, Including Number of Offers 

The results show no instances where source selection method (EPTA or tradeoff) 
produced significantly different contract outcomes. There were, however, covariates that 
produced different outcomes. Although not the focus of our research, we list them here 
and urge future researchers to examine these variables as covariates, as they capture some 
of the variance in the DV groups. Essentially, both the number of evaluation factors a 
solicitation contains and the number of offers it receives increase time to award (PAET). 
These results make sense because the PAET includes the time taken to select an awardee. 
This selection process involves evaluating each offer received in strict accordance with 
the evaluation factors set forth in the solicitation. Therefore, the selection process (and 
resulting PAET) will increase as more offers are submitted and as more evaluation 
factors are stipulated in solicitations. Table 2 shows the numerical evidence of this result. 
The italicized variables, NUMEVAEEACTORS and NUMOEEERS, were the only ones 
that demonstrated a statistically significant effect on PAET. 
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Table 2. MANCOVA Results with NUMOFFERS 


Variable 

Coeff 

StdErr 

t 

P>ltl 

95% Cl 

PALT 







EPTATO 

-.9154339 

.513016 

-1.78 

0.100 

-2.0332 

.2023319 

#Eval 

Factors 

3.456288 

1.10602 

3.12 

0.009*** 

1.046478 

5.866098 

# Reviews 

.0082172 

.4027713 

0.02 

0.984 

-.8693461 

.8857806 

# Ojfers 

.9032533 

.3112962 

2.90 

0.013** 

.2249971 

1.58151 

Constant 

1.507324 

1.058166 

1.42 

0.180 

-.7982225 

3.81287 

CPARS 







EPTATO 

.0447562 

.1424067 

0.31 

0.759 

-.2655213 

.3550337 

# Eval 
Eactors 

-.1412446 

.3070169 

-0.46 

0.654 

-.810177 

.5276879 

# Reviews 

.1214624 

.1118042 

1.09 

0.299 

-.122138 

.3650628 

# Offers 

-.0865176 

.0864119 

-1.00 

0.336 

-.2747929 

.1017576 

Constant 

1.413526 

.2937335 

4.81 

0.000 

.7735361 

2.053517 


* p<.10 ** p<.05 *** p<.01 

Number of Observations = 17 

Root Mean Squared Error: PAET = .86 CPARS = .24 

R^: PAET = .6266 CPARS = .1171 


DVs: PALT and CPARS ratings; statistically significant variables in italics; results 
shown in multivariate regression format 


2. Results, Excluding Number of Offers 

Removing the effect of the potentially collinear variable NUMOFFERS, we find 
similar results. There are still no instances where source selection method (EPTA or 
tradeoff) produced significantly different contract outcomes. Number of Evaluation 
Eactors (NUMEVAEEACTORS), however, still produced significantly different results 
in PAET (the larger the number of factors, the longer the PAET). This is shown by the 
italicized row in Table 3. 
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Table 3. MANCOVA Results without NUMOFFERS 


Variable 

Coeff 

StdErr 

t 

P>ltl 

95% Cl 

PALT 







LPTATO 

-.1955875 

.5627742 

-0.35 

0.734 

-1.411387 

1.020212 

#Eval 

Factors 

2.660771 

1.342889 

1.98 

0.069* 

-.2403637 

5.561905 

# Reviews 

.5984634 

.4356733 

1.37 

0.193 

-.3427515 

1.539678 

Constant 

1.921261 

1.31407 

1.46 

0.167 

-.9176158 

4.760138 

CPARS 







LPTATO 

-.0241939 

.1246599 

-0.19 

0.849 

-.2935053 

.2451175 

#Eval 

Lactors 

-.0650464 

.2974628 

-0.22 

0.830 

-.7076756 

.5775829 

# Reviews 

.0649259 

.0965058 

0.67 

0.513 

-.1435622 

.2734141 

Constant 

1.373878 

.2910793 

4.72 

0.000 

.7450391 

2.002716 


* p<.10 ** p<.05 *** p<.01 

Number of Observations = 17 

Root Mean Squared Error: PALT = 1.08 CPARS = .24 

R^: PALT = .3646 CPARS = .0433 


DVs: PALT and CPARS ratings; statistically significant variables in italics; results 
shown in multivariate regression format 


D. FURTHER DISCUSSION 

The previous two tables show that evaluation factors and number of offers 
received are the only variables that significantly affect contract outcome. These two 
variables affected PALT, even though the CPARS ratings were unaffected (significantly) 
by either the IVs or the covariates. As a final analysis, we examined the relationships 
between the variables themselves. The correlations are shown in Table 4. 
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Table 4. Correlations 


Variable 

PAET 

CPARS 

# Evaluation 
Eactors 

# Reviews 

# Offers 

PAET 

1.00 

-0.3623 

0.4694 

0.5344 

0.3333 

CPARS 

0.2657 

1.00 

0.1206 

0.3161 

-0.2882 

# Evaluation 
Eactors 

0.7319 

0.2258 

1.00 

0.5799 

-0.2951 

# Reviews 

0.3099 

0.0912 

-0.1722 

1.00 

0.3650 

# Offers 

0.9370 

0.4951 

0.6139 

0.4753 

1.00 


Non-italicized, lower triangle=LPTA; italicized, upper triangle=tradeoff 


These correlations confirm the previous findings, and should come as no surprise 
to anyone who has taken part in a source selection. For example, the highest overall 
correlation exists between PALT and NUMOFFERS (r=.9370), and between PALT and 
NUMEVALFACTORS (r=.7319) in an EPTA setting. This suggests that the bulk of the 
PAET is taken up by reviewing offers; the more offers that are submitted, the more time 
it will take to review those offers (thus increasing PAET). There is a high correlation 
between PAET and evaluation factors (second highest correlation found through 
correlation analysis). These two aforementioned high correlations demonstrate that there 
is potentially a large compounding effect of evaluation factors and number of offers on 
PAET. 

The highest correlation found in the tradeoff setting exists between evaluation 
factors and NUMREVIEWS (r=.5799). The same amount of strong correlation between 
these two covariates was not noticed in the EPTA situation. These results also make 
sense; for tradeoff scenarios, source selection is more complex as each evaluation factor 
is weighed as a possible basis for award. The more factors present as possible bases for 
award, the more complex this determination becomes; thus, the process will likely require 
more reviews. In contrast, with EPTA, the decision ultimately comes down to price 
(provided the lowest priced offer is technically acceptable), and the process is more 
straightforward, thus requiring fewer reviews. Other noteworthy correlations in the 
tradeoff setting include the correlation between PAET and NUMEVAEEACTORS 
(r=.4694) and also between PAET and NUMREVIEWS (r=.5344). These correlations 
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confirm that an increase in evaluation factors, and/or an increase in the number of 
reviews, will increase the PALT. The fact that this can be shown statistically should 
affirm the need to eliminate unnecessary levels of review or evaluation factors. Any 
increase to PALT should be for worthwhile evaluation. 

It is also worth noting the highest instance of negative correlation found in the 
analysis. This exists between the two DVs, PALT and CPARS (r=-0.3623). The possible 
reasons for this observation are numerous, but the following scenarios provide possible 
explanations. A low PALT could mean that the contract awardee was quickly selected as 
a clear winner and that this contractor lived up to expectations by performing well. 
Conversely, a high PALT suggests that the decision was more ambiguous, with no clear 
winner. A decision made under the latter conditions would be more likely to result in 
selecting a less-than-optimal contractor. While this is educated speculation, more 
research needs to be conducted to confirm or negate this notion. 

One last important observation from the correlation table is the positive 
correlation between number of offers and CPARS ratings in LPTA source selections 
(r=.4951). This correlation helps make the case that competition produces better 
outcomes. While nothing can be stated for certain here, this suggests that higher levels of 
competition will lead to the selection of high-performing contractors. 

E. RECOMMENDATIONS BASED ON FINDINGS 

We are unable to make concrete recommendations for change based on our 
limited amount of data. The one recommendation we can make is that each requirement 
be fully understood in order to make the best decision as to whether the LPTA or tradeoff 
source selection method is more appropriate. In the solicitation planning phase, 
contracting personnel together with their customers should closely examine the number 
of evaluation factors and the number of planned reviews, with the understanding that 
these variables will affect PALT. This recommendation mirrors that of contract 
management expert Marge Rumbaugh to “limit evaluation factors to those that are critical 
to program success” (Rumbaugh, 2015, p. 12). The likely additional PALT should be 
weighed against the need for certain evaluation factors and reviews to determine which is 
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more important to the acquisition. In other words, the acquisition risk posed by 
conducting fewer reviews should be weighed against that of increased PALT. These 
recommendations also echo the policy memorandum issued by Shay Assad, then-director 
of Defense Procurement and Acquisition Policy (2011). 

F. SUMMARY 

These results have served to mathematically confirm some basic assumptions of 
the source selection process. Also, the idea that one source selection method may lead to 
better outcomes than another was unsubstantiated by our analyses. We encourage further 
research efforts in this area to expand on these findings and possibly find other ways of 
predicting or improving contract outcomes. Having shown the results of our data 
analysis, in the next chapter, we draw conclusions and present recommendations, as well 
as answer our research questions. 
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V. SUMMARY, CONCLUSION, AND AREAS FOR FURTHER 

RESEARCH 


A. INTRODUCTION 

The previous chapter included a brief discussion of some of the observed effects 
of source selection method on contract outcomes, namely PALT and CPARS ratings. In 
this chapter, we summarize our findings and present our conclusion from the data 
analysis found in Chapter IV, answer the original research questions, and make 
suggestions for future research. 

B. SUMMARY 

The goal of this research was to determine whether or not there is a relationship 
between the source selection methods, and the contract outcomes of PALT and CPARS 
ratings. We reviewed many professional journal articles and Government Accountability 
Office reports to better understand both methods. We also performed hands-on research 
by reviewing 33 contract files at Peterson AFB and Schriever AFB and collecting the 
data points indicated in Chapters III and IV; then we utilized statistical analysis to 
determine the effects of those data points on PALT and CPARS ratings. Our data 
collection method produced a sample size that was adequate for our purposes but too 
small to influence policy decisions regarding source selection. We have made some 
suggestions regarding information provided by our analysis, with the recommendation 
that our study results be expanded upon and retested with future research. 

The main takeaway from our research is that PALT and CPARS are more affected 
by number of evaluation factors and number of offers received than they are by the 
source selection method used. The circumstances of each acquisition are different, and 
COs must exercise expert judgment in determining which source selection method to use. 
Both methods will prove successful as long as they are executed properly. Proper 
execution will work towards meeting the agency’s contracting needs and increasing 
mission success while also safeguarding limited government funds. 
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c. 


CONCLUSION 


The limitation of our research was that the sample size we collected was small; 
thus, our conclusions must be read with that caveat in mind. The data analysis showed 
that the source selection process used did not significantly affect PALT or CPARS 
ratings. Only the number of evaluation factors and the number of offers received 
produced a statistically significantly effect. Correlation analysis confirmed the finding 
that the number of evaluation factors and the number of offers were positively correlated 
to PALT, indicating that PALT increases along with an increase in either factor. 

D. ANSWERING THE RESEARCH QUESTIONS 

In Chapter I of this research, we posed the following questions: 

1. What is the relationship between the source selection method (LPTA 
versus tradeoff) and CPARS ratings? 

2. What is the relationship between the source selection method (LPTA 
versus tradeoff) and PALT? 

3. What percentage of procurements using the tradeoff source selection 
method is awarded based on non-price factors? 

Questions 1 and 2 can be answered concurrently. The underlying finding of the 
data analysis was that the source selection process used did not significantly affect PALT 
or CPARS ratings. Only the number of evaluation factors and the number of offers 
received significantly affected PALT. In other words, PALT will increase as complexity 
of the acquisition increases. This is not to say, of course, that evaluation factors should be 
minimized in order to limit PALT. If an acquisition is worth conducting, it is worth 
taking the time to do it correctly; there is no sense in sacrificing quality of source 
selection to reduce lead time. 

Contract performance, in the form of CPARS ratings, was not affected by any of 

the variables tested. This finding suggests that both source selection methods result in 

equal performance. From a common sense standpoint, this result makes sense, as there is 

no reason to suspect that contractors chosen with tradeoffs are inherently better than 

those chosen with the LPTA method—neither would the quality of contractor 

performance be expected to change as a result of variables such as the number of 
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evaluation factors. A contractor’s performance is not tied to the procedures the agency 
uses to select them for contract award. 

A reminder of our small sample size must accompany the answer to Question 3. 
Of the 23 contracts we sampled that used the tradeoff method, eight were awarded based 
on non-price factors. In other words, factors other than price were the basis for award for 
34.8% of our tradeoff sample contracts. The remaining 65.2% ended up being awarded 
based on price, even though the tradeoff method was used. This finding is in line with the 
Government Accountability Office’s finding, presented in our literature review; namely, 
that the government selected higher priced offers in less than a quarter of tradeoff 
examples observed (GAO, 2010). This is an interesting finding, although a much larger 
and varied sample size (i.e., from several installations) would be needed in order to draw 
any real conclusions about common practices. 

E. AREAS FOR FURTHER RESEARCH 

A consistent challenge throughout our project was ensuring we would have 
enough data points to ensure adequate power for our statistical analysis. Our limited 
sample size opens the door for future researchers to collect more data, in the same 
manner as our study, to re-run the model. Specifically, we recommend that further 
research of this nature be conducted at other operational bases within the Air Force. From 
there, the research should expand to major weapon system acquisition bases. After that 
research has been conducted, the other military branches should duplicate these research 
efforts, both at the operational- and systems-levels. As stated previously, our hope is that 
this research serves as a starting point for a larger effort that eventually encompasses the 
entire defense acquisition field. With such a large data sample, the model will obtain 
adequate statistical power. Such a development would allow future researchers to draw 
more accurate and definitive conclusions as to the relationship between source selection 
method and contract outcomes. At that point, the findings may point to areas where the 
source selection process can be improved and cost or process savings realized across the 
Department of Defense. 
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